Fused imaging device and method

ABSTRACT

A fused imaging device and method are disclosed. The device comprises a light source component, an image capture component, and a control component. The control component may be configured to control the light source component to illuminate a target object based on a preset plurality of first optimal lighting configurations. The control component may further control the image capture component to capture images of the target object to obtain multiple first images, under the illumination of the light source component and generate a target image of the target object, based on the first images. The control component may further adjust the incidence angle, pattern, and wavelength of the light source in the light source component, as well as the exposure, lens focus, and polarization of the image capture component, to detect target object defects in captured images of the target object.

TECHNICAL FIELD

The present disclosure relates to the field of machine vision, and inparticular to a fused imaging device and method.

BACKGROUND

In industrial production, image recognition is used to detect surfacedefects on products, which include, for example, metal castings.However, when capturing images, a particular light source is frequentlynot conducive for imaging a product having a particular surface, aparticular defect, and/or a particular environment. For example, surfacematerials or characteristics on products that affect light quality of acaptured image include reflective qualities, transparent qualities, orblack/opaque qualities. In another example, a particular light sourcemay not be compatible with imaging multiple types of defects including,for example, scratches or dirt. In another example, a particular lightsource may not be compatible with creating satisfactory images invarious environments, such as laboratories and production lines. Thelight quality of captured images directly affects the ability to detectsurface defects on products and may result in reduced accuracy forcorrectly identifying such defects.

SUMMARY OF THE DISCLOSURE

Disclosed below is a device which comprises a light source component, animage capture component, and a control component. The light sourcecomponent includes a plurality of light sources disposed on an interiorsurface of a shell of the light source component.

Further disclosed below is a method comprising illuminating, by a lightsource, a target object with a plurality of lighting conditionsaccording to a preset multiple first optimal lighting configuration. Themethod further includes capturing, by an image sensor, an image of thetarget object in each of the plurality of lighting conditions to obtaina plurality of images. The method further includes generating, by aprocessor, a target image of the target object from the plurality ofimages. The method further includes detecting, by a processor, a surfacedefect on the target object based on the target image. The methodfurther includes identifying, by a processor, a location on the targetobject of the surface defect. The method also includes providing, by aprocessor, a visual indication to a user of the location of the surfacedefect on the target object.

BRIEF DESCRIPTION OF THE DRAWINGS

The following gives a detailed description of the specific embodimentsof the disclosure, accompanied by diagrams to clarify the technicalsolutions of the embodiments and their benefits.

FIG. 1 illustrates a block diagram of a fused imaging device based on anembodiment of the present disclosure.

FIGS. 2 a and 2 b illustrate schematic cross-sectional views of ahemisphere according to an embodiment of the present disclosure.

FIGS. 3 a to 3 g illustrate schematic diagrams of the light sourcearranged as cross-sections inside a light housing according to anembodiment of the present disclosure.

FIGS. 4 a to 4 g illustrate schematic diagrams of the light sourcearranged as a geodesic dome inside the light housing according to anembodiment of the present disclosure.

FIG. 5 illustrates a flow diagram of a method executed by a controlcomponent according to an embodiment of the present disclosure.

FIG. 6 illustrates a flow diagram of Step S503 of the control component,shown in FIG. 5 and according to an embodiment of the presentdisclosure.

FIG. 7 illustrates a flow diagram of a fused imaging method according toan embodiment of the present disclosure.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present invention willbe clearly and completely described below, accompanied by diagrams ofembodiments. The described embodiments are only an explanatory andillustrative part of many embodiments consistent with the scope of thisdisclosure and is intended to describe conceptual implementations forpurposes of informing one of ordinary skill in the art with noveldevelopments to the art described herein in lieu of disclosing allpossible embodiments. Based on the embodiments of the present invention,all other embodiments obtained by those of ordinary skill in the artwithout creative work shall fall within the scope of the presentdisclosure.

Manufacturing currently faces many obstacles regarding the light sourcesused for capturing images of industrial products. For example, no singleexisting light source provides adequate light for capturing an imagewith quality sufficient to ascertain the existence of surface defects onall surface materials. For different surface materials, such asmaterials with as reflective, transparent, or black surfaces,wavelengths and patterns of light emitted by a light source to capturehigh-quality images have been conventionally necessary for adequateimage quality. Further, no conventional single light source providesadequate lighting for image capture with multiple types of defects orwith different types of defects such as scratches, dirt, and defectswith random locations, sizes, or shapes. Such imagery has conventionallyrequired customized light sources with specific incidence angles andpatterns. Finally, no conventional light source has been adequate toprovide adequate lighting for image capture in all environments, becausedifferent environments, such as laboratories and production lines) aretypically large areas and which produce products that are made fromdifferent materials. Thus, capturing images with adequate quality fordetecting defects has relied on manual manipulation of distance andgeometry between lighting, a camera, and the product to capturehigh-quality images.

The present disclosure proposes a fused imaging device. By usingcustomizable first optimal lighting configurations, adjusting incidenceangles, patterns, and wavelength of light emitted by a light source inthe light source component, the light source component may adequatelyilluminate a product for high quality image capture in many conditions,such as multiple surface materials, defect types, and environments.Under the illumination of the light source component, the controlcomponent may automatically adjust the exposure, lens focus, andpolarization of the image capture component, and control the imagecapture component to capture multiple images of the target object, andthen generate the target image using the multiple collected images,thereby improving the accuracy defect detection using the image.

FIG. 1 illustrates a block diagram of a fused imaging device 100according to an embodiment of the present disclosure. As shown in FIG. 1, the fused imaging device 100 includes a light source component 110,which illuminates a target object, such as a metal casting or otherproduct. Light source component 110 comprises a light source and a lighthousing. The light source within light source component 110 comprises aplurality of LEDs and/or flexible screens. Fused imaging device 100 alsoincludes an image capture component 120, which captures images of thetarget object. Image capture component 120 may be implemented by acamera with an image (e.g., optical or light) sensor which receiveslight, and creates an image based on light received into the imagesensor. Fused imaging device may further include a control component130, which is connected to the light source component 110 and the imagecapture component 120.

Control component 130 may be implemented by a hardware processordisposed within fused imaging device 100. Control component 130 mayfurther include hardware components which may include a combination ofCentral Processing Units (“CPUs”), buses, volatile and non-volatilememory devices, storage units, non-transitory computer-readable media,data processors, processing devices, control devices transmitters,receivers, antennas, transceivers, input devices, output devices,network interface devices, and other types of components that areapparent to those skilled in the art. These hardware components withinthe user device may be used to execute the various applications,methods, or algorithms disclosed herein independent of other devicesdisclosed herein. Control component 130 may execute softwareinstructions, sequences of instructions, routines, data structures,display interfaces, and other types of structures that execute computingoperations. Control component 130 may, for example, control the lightsource component 110 to illuminate a target object. Control component130 may illuminate the target object according to a plurality of presetfirst optimal lighting configuration which includes a lightingconfiguration matrix, where each value of the lighting configurationmatrix represents a working status of each independently controllableLED and/or each pixel on a flexible screen. The working status may beindicated using an indicator representing at least one of an off state,an on state, or a value of brightness.

Control component 130 may further control the image capture component tocapture multiple images of a target object to obtain multiple firstimages, under the illumination of the light source component. Once thesemultiple first images are obtained, control component 130 may generate atarget image of the target object based on the plurality of firstimages.

In another embodiment, light source component 110 may illuminate atarget object. The light source component 110 may comprise a lightsource and a light housing, and the light source may comprise a numberof LEDs (light emitting diodes) and/or flexible screens. The lightsource may also comprise a set of other LEDs that may be independentlycontrolled. The present disclosure does not limit the specific types oflight sources. In another embodiment, when the light source is multipleLEDs, the multiple LEDs may be formed into the light housing as aPrinted Circuit Board (PCB) by either surface mounts or directinsertions.

Colors (e.g., wavelengths) of the light emitted by the LEDs may be thesame, substantially the same, or different. For example, the LEDs mayemit white, red, blue, green lights or even infrared and ultraviolet.Different colors or types of the light emitted by the LED are emitted atdifferent wavelengths of light according to known principles of opticalphysics. Those skilled in the art may select LEDs with differentwavelengths/colors according to specific conditions that are appropriatefor a particular circumstance and is not to be considered limited bythis disclosure.

In another embodiment, it is possible to control the wavelength of thelight source, based on the principle of color mixing (e.g., combiningred and green light sources to produce yellow light) by controlling thebrightness (i.e., current) of multiple LEDs; it is also possible tocontrol the pattern and incidence angle of the light source, bycentralized placement of multiple LEDs and independent control, bycontrol component 130, of all or some of the LEDs including the abilityto turn lights on, to turn lights off, or to adjust a brightness ofLEDs.

For example, assuming that there are 64 LEDs installed in a lighthousing implemented in lighting component 110, the 64 LEDs may becompactly packed in an 8 LED by 8 LED square, where each LED is anindependently controlled channel. This allows fine control of the 64-LEDlight source, giving control component 130 the ability to adjust thelight source's pattern and/or incidence angle. The pattern and/orincidence angle of the light source may also be adjusted by controllingsome of the 64 LEDs (for example, 48 or 32 LEDs). The present disclosuredoes not limit the number of controllable LEDs.

In another embodiment, when the light source is implemented as flexiblescreens, the light source may be implemented as a whole screen or maycomprise multiple sub-screens. The flexible screens may also be formedinside the light housing of lighting component 110. The flexible screensmay be flexible LCD screens, LED screens, OLED screens, micro-LEDscreens, or other types. The present disclosure does not limit thespecific type of flexible screens.

In another embodiment, all or some pixels in the flexible screen maylikewise be independently controlled by control component 130 includingthe ability to turn a light on, to turn a light off, or to adjust abrightness of one or more pixels. Control component 130 may also controla wavelength, pattern, and incidence angle of the light source in amanner similar to that discussed above with respect to using LEDs in thelight source. However, compared with LEDs, the resolution of theflexible screen is higher, allowing for finer and more accurate controlof the light source.

In another embodiment, the light housing within light source component110 may be implemented approximately in the shape of a hemisphericalshell as will be discussed below. The light source may be set ascross-sections, geodesic dome, or hemispherical shell inside the lighthousing, such that the vertical and/or the horizontal incidence angle ofthe light from the light source may be controlled by control component130.

FIGS. 2A and 2B illustrate schematic cross-sectional views of ahemispherical shell 200, discussed above and according to an embodimentof the present disclosure. As shown in FIG. 2A, hemispherical shell 200is illustrated as a vertical cross-section of a hemisphere. Multipleradii extending to α1, α2 . . . α10 are illustrated as emanating from acenter of the hemisphere 205 to an edge of hemisphere 205 where thelabels of α1, α2 . . . α10 are shown on the page. Each of radii α1, α2 .. . α10 are disposed such that the angle between neighboring radii(e.g., α1 and α2, α2 and α3, etc.) is set at a first preset value. Asused herein, the point at which each of radii α1, α2 . . . α10 meet anedge of the hemispherical shell 200 is referred to as intercept or anintersection. Intercepts that are neighboring (or adjacent) may each bejoined by line segments such that these multiple line segments form anapproximate arc that follows the edge of hemispherical shell 200. Eachintercept may be extended in the horizontal plane to divide thehemisphere into multiple cross-sections 210 where a light source may beinstalled in the ring on each cross-section.

In another embodiment, different cross-sections 210 may form differentvertical incidence angles so that the vertical incidence angle may becontrolled by control component 130 controlling each light source in thecross-sections. As the vertical incidence angle changes, the anglebetween the surface of the target object and the x-y plane (using theillustrated cartesian plane references) varies to give differentreflections, achieving different effects in image capturing.

FIG. 2B illustrates a schematic diagram of one of a plurality ofhorizontal cross-sections 210 of the hemisphere. As shown in FIG. 2B,starting from the center of the hemisphere 205, multiple radii extendingto β1, β2 . . . β16; may be identified such that the angle betweenneighboring radii is set at a second preset value, and each radius meetsthe arc of the hemisphere at an intercept (i.e., intersection), labeledas β1, β2 . . . β16. Each neighboring (or adjacent) intercept may bejoined by to the center of hemisphere 205 with line segments which maythen be extended vertically into a plane to divide the hemisphere intomultiple vertical sections 215 (which identifies a vertical sectionbetween β2 and β3 but intended to refer collectively to any verticalsection which exists between any two neighboring intercepts β1, β2 . . .β16).

In another embodiment, different vertical sections 215 may be formed atdifferent horizontal incidence angles such that a vertical incidenceangle may be controlled by controlling the light source on the verticalsections 215. Along a z-axis (as referenced in a cartesian sense usingthe illustrated cartesian plane representation), a surface of a targetobject may produce different reflections as the horizontal incidenceangle of the vertical sections 215 change so as to achieve differentimage capturing effects.

In another embodiment, U (horizontal) cross-sections may be intersectedwith V vertical sections (U and V are both positive integers) to formU×V light-emitting areas, and each light area may be individuallycontrolled by control component 130. Thus, both vertical and horizontalincidence angles may be independently controlled, allowing for betterimage capture.

In another embodiment, the light source may be implemented as aplurality of cross-sections, a geodesic dome, or hemispherical shellinside the light housing, according to the above-mentioned verticaland/or horizontal incidence angle. For example, the light housing may bedivided into cross-sections, with each section having a light sourcesuch as one or more LEDs or other lights attached to flexible materials.Alternatively, the light housing may be an approximately geodesic domecomprised of a plurality of individual small triangles, with each smalltriangle having a piece of flexible screen or other rigid material witha rigid light-emitting source attached. When the light source is theentire flexible screen, the flexible screen may be bent into ahemisphere and formed into the light housing within light sourcecomponent 110.

FIGS. 3A to 3G show schematic diagrams of a light source 300 of lightsource component 120, shown in FIG. 1 arranged as cross-sections insidethe light housing according to an embodiment of the present disclosure.FIGS. 3A-3D show side view, top view, oblique view, and bottom view,respectively. FIG. 3D in particular shows a hemispherical area withinlight source 300 within which an object to be imaged may be positionedfor appropriate lighting, using the techniques described herein, andimaging. These objects being imaged are intended to fit within lightsource 300. Further, light source 300 may include connections to cablesfor receiving and transmitting power or information, as appropriate. Aplurality of LEDs 305 is arranged in vertical and horizontalcross-sections of the light housing as shown in FIG. 3E. FIGS. 3F and 3Gshow a bottom view and a side view of a single horizontal cross section210 (shown in FIG. 2A).

It should be noted that although FIGS. 3A-3G use LEDs as examples of thelight source 300, those skilled in the art should understand that thelight source is not limited to LEDs and may be implemented be othersources which are known to those of ordinary skill in the art ordescribed herein.

FIGS. 4A-4G illustrate schematic diagrams of light source 400 of lightsource component 120, shown in FIG. 1 , implemented as a geodesic domein a light housing according to an embodiment of the present disclosure.FIGS. 4A-4E illustrate a show side view, top view, oblique view, bottomview, and cross-section of the light source 400 arranged as a geodesicdome inside the light housing. FIGS. 4F and 4G illustrate a front viewand an oblique view of a single triangle 405 when the light source 400is arranged as a geodesic dome inside the light housing.

It should be noted that although FIGS. 4F and 4G use LEDs 410 asexamples of the light source, those skilled in the art should understandthat the light source is not limited to LEDs and may be implemented beother sources which are known to those of ordinary skill in the art ordescribed herein.

It should be noted that those skilled in the art may select thearrangement of the light source and the light housing based on thespecific shapes of both. Specific implementations are not limited tomerely what is discussed here but, rather, extends to thoseimplementations that embody the scope and spirit of the disclosure.

Based on the vertical and/or the horizontal incidence angle of the raysof light emitted by light source 300/400, the light source 300/400 maybe implemented as cross-sections, a geodesic dome, or a hemisphericalshell inside the light housing of light source component 110, such thatthe vertical and/or horizontal incidence angle of the light source maybe controlled to achieve better image capture.

In another embodiment, the image capture component 120 may be used tocapture images of the target object. The image capture component maycomprise image-capturing devices, such as cameras and video cameras. Thecontrol component 130 may automatically adjust or set parameters such asthe exposure time, lens focus, and polarization of the image capturecomponent 120, using any technique known in the art.

In another embodiment, the control component 130 may be connected to thelight source component 110 and the image capture component 120. Thecontrol component 130 may include a processor which may control thelight source component to illuminate the target object according to aplurality of preset first optimal lighting configuration. The firstoptimal lighting configuration may comprise a lighting configurationmatrix, with each value of the lighting configuration matrixrepresenting a working status of each independently controllable lightelement, whether an LED or each pixel of a flexible screen. The workingstatus may include at least one of an off state, an on state, or a valueof brightness.

The processor may further control the image capture component 120 tocapture multiple images of the target object to obtain multiple firstimages, under the illumination of the light source component, andgenerate a target image of a target object based on the plurality offirst images.

In another embodiment, the preset multiple first optimal lightingconfigurations may be set randomly or based on experience. It is alsopossible to use automatic iterative testing to find the optimal lightingconfiguration and use this optimal configuration to determine aplurality of first optimal lighting configurations. The presentdisclosure does not limit the specific method of determining the firstoptimal lighting configuration.

In another embodiment, the first optimal lighting configuration may beexpressed as a lighting configuration matrix. Each value of the lightingconfiguration matrix represents the working status of each independentlycontrollable LED and/or each pixel on the flexible screen, with theworking status being at least one of an off condition, an on condition,or a value of brightness. For example, a value of 0 in the lightingconfiguration matrix means that particular light element (LED or pixel)is off, a value greater than 0 means on, a value of 1 means the maximumbrightness, and a value of 0.5 means that the brightness is half of themaximum brightness.

In another embodiment, when the LEDs or the pixels of the flexiblescreen have multiple wavelengths/colors, independent control channelsmay be set for each wavelength/color, with a corresponding row or columnin the lighting configuration matrix.

For example, an LED may emit light in three primary colors, red, green,and blue. Three independent control channels may be set for this LED,each corresponding to one color, and the lighting configuration matrixmay have a row or column corresponding to each channel to blend the red,green, and blue color to produce another visible color. It should benoted that LEDs that emit light in the UV or infrared range may besimilarly controlled.

In another embodiment, the control component 130 may control the lightsource component 110 to illuminate the target object according to themultiple first optimal lighting configurations, and under thisillumination, the control component 130 may control the image capturecomponent 120 to capture multiple images of the target object and obtainmultiple first images.

Light source component 110 may include multiple illumination modescorresponding to the first optimal lighting configurations. In anylighting mode, the control component 130 may control the image capturecomponent 120 to perform one image capture of the target object toobtain the first image, and order the light source component 110 toswitch to another lighting mode and perform image capture under this newmode until one image is captured under all lighting modes. These imagesform a set of first images.

During the image capture process, each object (including the lightsource component 110, image capture component 120, and target object)may be stationary, meaning that the relative positions and anglesbetween all objects remain unchanged.

In another embodiment, after obtaining a plurality of first images, thecontrol component 130 may generate a target image of the target objectby division, selection, and fusion using the plurality of first images.

According to an embodiment of the present disclosure, the fused imagingdevice comprises a light source component 110, an image capturecomponent 120, and a control component 130. Based on the preset multiplefirst optimal lighting configurations, the control component 130 maycontrol the light source component 110 to illuminate the target object.Under such illumination, the control component 130 may control the imagecapture component to capture multiple images of the target object toobtain multiple first images and generate the target image of the targetobject according to these first images. By using a plurality ofcustomizable first light configurations and adjusting incidence angles,patterns, and wavelength of the light source in the light sourcecomponent, the light source component 110 may be made compatible withmultiple surface materials, defect types, and environments. By capturingimages multiple times and using these images to generate the targetimage, the method ensures that the target image is globally optimal inproviding an accurate image of the target object.

In another embodiment, generation of the target image of the targetobject using the multiple first images comprises may be completed by aprocessor in control component 130 by at least one of the followingsteps: (1) determining, respectively, the optimal target pixels for eachpixel position, based on the characteristic value of each pixel at thesame pixel position in the plurality of first images, and generate atarget image of the target object according to the plurality of targetpixels; 2) selecting a plurality of characteristic areas from theplurality of first images according to preset selection criteria, andgenerating a target image of the target object according to theplurality of characteristic area; and 3) generating a three-dimensionalimage of the target object according to the plurality of the firstimages, and set this three-dimensional image as the target image of thetarget object.

In another embodiment, in generating the target image, it is possible todetermine, respectively, the optimal target pixels for each pixelposition, based on the characteristic value of each pixel at the samepixel position in the plurality of first images, and generate a targetimage of the target object according to the plurality of target pixels.

Any pixel position may have many pixels, each from a different firstimage at the corresponding position. From these pixels, it is possibleto pick the pixel with the optimal characteristic value and set it asthe target pixel and generate the target image by using the multipledetermined target pixels.

An optimal characteristic value may include obtaining a largestgrayscale value (brightest), a smallest grayscale value (darkest), alargest convolutional characteristic value, and a smallest convolutionalcharacteristic value. The convolutional characteristic value may beobtained by performing convolution on a convolutional area, which may bedefined as a set of pixels adjacent to one particular pixel. The weightof the convolutional area in obtaining an optimal characteristic valuemay be determined by manual commissioning or automatic learning (machinelearning, deep learning, and/or by use of artificial intelligence).

It should be noted that those skilled in the art may determine aspecific criteria for choosing the optimal characteristic value withoutdeparting from the scope or spirit of this disclosure.

In another embodiment, in generating the target image from multiplefirst images, it is possible to select multiple characteristic areasfrom the first images according to a preset criteria and generate atarget image of the target object from these characteristic areas. Areaselection criteria may be set according to actual conditions. Forexample, the selection criteria may comprise any one or more of thefollowing: 1) selecting areas in the first images where thecharacteristic values of all pixels are greater than or equal to apreset first threshold; 2) selecting areas in the first images where thecharacteristic values of all pixels are less than or equal to a presetsecond threshold; and 3) selecting the area with the highest recognitionaccuracy in the first images; where recognition accuracy may meansbrightness and sharpness; for example, select pixels with brightness inthe range of unsaturated grayscale zone, or select pixels with thehighest sharpness; and 4) selecting the area with the highestconvolutional characteristic value in the first images.

It should be noted that the area selection criteria may also includeother conditions, and the present disclosure is not exhaustive ofspecific area selection criteria.

In another embodiment, when generating the target image of the targetobject based on multiple first images, it is possible to generate athree-dimensional image based on the multiple first images throughimaging principles, for example, using the parameters of the lightsource component and the image capture component. This three-dimensionalimage may be used as the target image.

In this embodiment, generating pixel points with the optimalcharacteristic values in the multiple first images may be selected, ormultiple characteristic areas may be selected from the multiple firstimages, or the three-dimensional images of the target object may begenerated to determine a globally optimal image the target object. Usinga globally optimal image may improve the accuracy of the target image.

In another embodiment, the control component 130 may cause the processorto 1) obtain N lighting configurations from the preset set of lightingconfigurations, by using N times of optimization selections, where N isan integer and N≥2; 2) determine the scores of the N−1^(th) and theN^(th) lighting configurations; 3) determine the second optimal lightingconfiguration based on the scores of the N−1^(th) and the N^(th)lighting configurations; and 4) determine a plurality of first optimallighting configurations based on the second optimal lightingconfiguration.

In another embodiment, the number of lighting configurations in thepreset lighting configuration set may be very large, for example, 10⁸⁰or more. To select the optimal lighting configuration from the lightingconfiguration set, an approximation may be used to select N lightingconfigurations by conducting N optimization selections.

After selecting N lighting configurations, scores for each of theN−1^(th) and the N^(th) lighting configurations may be determined,respectively. Of which, a score of the N−1^(th) lighting configurationmay be the score of the N−1^(th) image captured according to theN−1^(th) lighting configuration; the score of the N^(th) lightingconfiguration is the N^(th) image captured according to the N^(th)lighting configuration.

A second optimal lighting configuration may be determined according tothe scores of the N−1^(th) and the N^(th) lighting configurations. Forexample, when the score of the N^(th) lighting configuration is greaterthan the score of the N−1^(th) configuration, the N^(th) configurationis set as the second optimal lighting configuration. Otherwise, theN−1^(th) lighting configuration is set as the second optimal lightingconfiguration. The determined second optimal lighting configuration maybe seen as the approximately optimal lighting configuration among theentire set of possible configurations. Then, the second optimal lightingconfiguration may be adjusted multiple times to obtain a plurality ofthe first optimal lighting configurations.

In this embodiment, through N optimization selections, N lightingconfigurations may be selected from the lighting configuration set, andthe second optimal lighting configuration may be determined based on thescores of the N−1^(th) and the N^(th) lighting configurations. Then, aplurality of first optimal lighting configurations may be determinedaccording to the second optimal lighting configurations, improving theefficiency and accuracy of selecting the first optimal lightingconfigurations.

In another embodiment, the control component 130 may obtain N lightingconfigurations from the preset set of lighting configurations, by usingN times of optimization selections to 1) select any lightingconfiguration from the preset set of lighting configurations as the 1stlighting configuration; 2) determine the 2nd lighting configurationbased on the 1st lighting configuration and preset selection criteria;3) determine the currently optimal lighting configuration based on thei−1^(th) and the i^(th) lighting configurations, with i being an integerand 2≤i≤N−1; and 4) determine the i+1^(th) lighting configuration basedon the currently optimal lighting configuration and the selectioncriteria.

In another embodiment, when performing N optimization selections, alighting configuration may be randomly selected from the preset lightingconfiguration set as the 1st lighting configuration; then, the 2ndlighting configuration may be determined based on the 1st lightingconfiguration and preset selection criteria.

Preset selection criteria may represent how to determine the lightingconfiguration matrix of the adjacent lighting configuration (forexample, the 2nd lighting configuration) according to the matrix of thecurrent lighting configuration (in this case, the 1st lightingconfiguration). The selection criteria may comprise any of thefollowing:

1. determining the lighting configuration matrix of the adjacentlighting configuration by transforming a single value in the lightingconfiguration matrix of a current lighting configuration, which mayspecifically comprise: when the lighting configuration matrix of thecurrent lighting configuration is not all 0, the transformed value isnot 0; or, when the data in the lighting configuration matrix of thecurrent lighting configuration is all 0, the transformed value may beany value within the possible range;

2. determining the lighting configuration matrix of the adjacentlighting configuration by transforming a single value in the lightingconfiguration matrix of the current lighting configuration, which mayspecifically comprise: setting all values in a certain area in thelighting configuration matrix of the current lighting configuration(with all values being non-zero) to the same new value;

3. determining the lighting configuration matrix of the adjacentlighting configuration by performing operations such as translation,rotation, scaling, stretching, reflection, and projection on thelighting configuration matrix set by the current light source (forexample, when performing translation, the lighting configuration matrixset by the current light source may be translated by a certain distancein the horizontal or vertical direction to obtain the lightingconfiguration matrix of the adjacent light configuration. When a row orcolumn that comprises a value other than 0 is removed from the lightingconfiguration matrix, the removed row or column may be added to theopposite row or column); and

4. other two-dimensional transformations of the lighting configurationmatrix.

It should be noted that those skilled in the art may set appropriateselection criteria according to actual conditions, and the presentdisclosure does not limit the choices.

In another embodiment, the currently optimal lighting configuration maybe determined according to the i−1^(th) and the i^(th) lightingconfigurations, and then the i+1^(th) light source may be determinedaccording to the currently optimal lighting configuration and selectioncriteria. In this way, N lighting configurations may be obtained.

In another embodiment, a method that determines the currently optimallighting configuration based on the i−1^(th) lighting configuration andthe i^(th) lighting configuration comprises: 1) determining the scoresof the i−1^(th) and the i^(th) lighting configuration, respectively; and2) when the score of the i^(th) lighting configuration is greater thanthe score of the i−1^(th) lighting configuration, set the i^(th)lighting configuration as the currently optimal lighting configuration.

In another embodiment, scores of the i−1^(th) and the i^(th) lightingconfigurations may be determined independently.

When determining the score of the i−1^(th) lighting configuration, thecontrol component may control the light source component 110 toilluminate the target object according to the i−1^(th) lightingconfiguration; under the illumination of the light source component 110,the image capture component 120 captures images of the target object toobtain the i−1^(th) image; then, according to the preset defect area,the control component 130 may mark defects on the i−1^(th) image todetermine the defect area in the i−1^(th) image.

After determining the defect area in the i−1^(th) image, the pixels inthe defective area in the i−1^(th) image may be regarded as multiplefirst pixels, and the first average value of the first pixel values maybe determined; the pixels outside the defect area in the i−1^(th) imagemay be regarded as second pixels, and the second average value of thepixel values of the second pixels may be determined; then. The controlcomponent 130 may further calculate a difference between the first andsecond average value, and take a ratio of this difference to a presettheoretical maximum difference (for example: if the pixel value is the0-255 grayscale, then the theoretical maximum difference is set to 255)as a score of the i−1^(th) lighting configuration.

In another embodiment, when the i−1^(th) image includes multiplechannels, the multiple first pixels comprise pixels in the defect areasof all channels of the i−1^(th) image; the multiple second pixelscomprise pixels outside the defect area of all channels of the i−1^(th)image.

In another embodiment, scores of the i−1^(th) lighting configuration maybe determined by the following equation:

$S = \frac{❘{\overset{\_}{P_{ng}} - \overset{\_}{P_{ok}}}❘}{P_{\max}}$

where, P_(ng) represents a first average value of pixels, P_(ok)represents a second average value of pixels, and P_(max) represents thetheoretical maximum difference.

In another embodiment, a method for determining a score of the i^(th)lighting configuration may be similar to that for the i−1^(th) lightconfiguration.

In another embodiment, after obtaining scores of the i−1^(th) and thei^(th) lighting configurations, the two may be compared. When the scoreof the i^(th) lighting configuration is greater than the score of thei−1^(th) lighting configuration, the i^(th) lighting configuration isdetermined as a currently optimal lighting configuration. Then, acurrently optimal lighting configuration may be used as the center ofthe search, and the i+1^(th) lighting configuration may be determinedfrom the lighting configuration set according to the selection criteria.

In another embodiment, determining the currently optimal lightingconfiguration according to the i−1^(th) and the i^(th) lightingconfigurations may further comprise: 1) when a score of the i^(th)lighting configuration is less than or equal to the score of thei−1^(th) lighting configuration, determining a probability of choosingthe i^(th) lighting configuration as the currently optimal lightingconfiguration, based on the scores of the i^(th) and the i−1^(th)lighting configurations, and the selection round i; and 2) when aselection probability is greater than the preset probability threshold,the i^(th) lighting configuration is set to be the currently optimallighting configuration.

In another embodiment, in the case that a score of the i^(th) lightingconfiguration is less than or equal to the score of the i−1^(th)lighting configuration, it may be advisable to determine a selectionprobability of selecting the i^(th) lighting configuration as acurrently optimal lighting configuration. Based on scores of the i^(th)and the i−1^(th) lighting configurations and the selection round i, aprobability of selecting the i^(th) lighting configuration as thecurrently optimal lighting configuration may be determined.

In another embodiment, the following equation may be used to determinethe probability p_(i) of selecting the i^(th) lighting configuration asthe currently optimal lighting configuration:p _(i) =e ^((S) ^(i) ^(−S) ^(i-1) ^()/T)

where S_(i) represents the score of the i^(th) lighting configuration,S_(i-1) represents the score of the i−1^(th) lighting configuration,T=(N−i)/N, and e is the base of the natural logarithm.

In another embodiment, after determining the selection probability ofselecting the i^(th) lighting configuration as a currently optimallighting configuration, a selection probability, and a preset selectionprobability threshold may be compared. In the case that a selectionprobability of selecting the i^(th) lighting configuration as thecurrently optimal lighting configuration is greater than the selectionprobability threshold, the i^(th) lighting configuration may beconsidered to meet the selection criteria, and the i^(th) lightingconfiguration may be determined as the currently optimal lightingconfiguration.

In another embodiment, determining a currently optimal lightingconfiguration according to the i−1^(th) and the i^(th) lightingconfigurations may further comprise: when a selection probability isless than or equal to a selection probability threshold, an i−1^(th)lighting configuration is determined as a currently optimal lightingconfiguration.

As a result, when the selection probability of selecting an i^(th)lighting configuration as the currently optimal lighting configurationis less than or equal to a selection probability threshold, it may beconsidered that an i^(th) lighting configuration does not meet aselection criteria, and the i−1^(th) lighting configuration isdetermined as a currently optimal lighting configuration.

In another embodiment, in determining a second optimal lightingconfiguration according to scores of the N−1^(th) and the N^(th)lighting configurations, when a score of the N^(th) lightingconfiguration is less than or equal to a score of the N−1^(th) lightingconfiguration, the above method may also be used to determine aprobability of selecting an N^(th) configuration as a currently optimallighting configuration. When that probability is greater than aselection threshold, the N^(th) light source is set to be a secondoptimal lighting configuration and when that probability is less than orequal to a selection probability threshold, the N−1^(th) lightingconfiguration may be set to be the second optimal lightingconfiguration.

FIG. 5 shows a flow diagram of how the control component 130 may use aprocessor to execute a method according to an embodiment of the presentdisclosure. As shown in FIG. 5 , when capturing images of a targetobject, multiple first optimal lighting configurations may be determinedaccording to an optimal lighting configuration (i.e., the second optimallighting configuration) selected from the preset lighting configurationset. At step S501 a processor selects any lighting configuration fromthe set of lighting configurations as a 1st lighting configuration. Atstep S502 the processor determines a 2nd lighting configurationaccording to the 1st lighting configuration and selection criteria, andat step S503 the processor determines a currently optimal lightingconfiguration according to the i−1^(th) and the i^(th) lightingconfigurations. For example, a currently optimal lighting configurationmay be determined according to the 1st and 2nd lighting configurations.At step S504 the processor determines an i+1th lighting configurationaccording to the currently optimal lighting configuration and selectioncriteria, and at step S505 the processor determines whether or not iequals N−1, that is, whether i==N−1.

When i does not equal to N−1, at step S511, a processor increases thecount of i by 1, and then continues to execute steps S503 through S505until i equals N−1;

When i does equal N−1, in step S506, the second optimal lightingconfiguration may be determined by the processor according to theN−1^(th) and the N^(th) lighting configurations.

After determining the second optimal lighting configuration, theprocessor at step S507 determines multiple first optimal lightingconfigurations according to a second optimal lighting configuration. Atstep S508 the processor controls the light source component toilluminate the target object according to the multiple first optimallighting configurations. At step S509 the processor controls imagecapture component 120 to capture multiple images of the target objectunder the illumination of the light source component 110, to obtain aplurality of first images. At step S510 the processor generates a targetimage of a target object, based on the multiple first images. In thisway, the steps completed the fused imaging process. The term “fused” inthis context refers to the joining of optimal lighting configurationsacross a plurality of image to produce an optimal image of a targetobject.

FIG. 6 illustrates a flow diagram of step S503 of the control component130 according to an embodiment of the present disclosure. As shown inFIG. 6 , when determining an i+1^(th) lighting configuration accordingto a currently optimal lighting configuration and selection criteria, Atstep S5031, a processor determines the scores of the i−1^(th) and thei^(th) lighting configurations, respectively. At step S5032 theprocessor determines whether a score of the i^(th) lightingconfiguration is greater than a score of the i−1^(th) lightingconfiguration.

When a score of the i^(th) lighting configuration is greater than ascore of the i−1^(th) lighting, step S5033 is performed by the processorwhich sets the i^(th) lighting configuration as a currently optimallighting configuration.

When a score of the i^(th) lighting configuration is less than or equalto a score of the i−1^(th) lighting configuration, step S5034 isperformed by the processor to determine a probability of selecting thei^(th) lighting configuration as the currently optimal lightingconfiguration. At step S5035, the processor judges whether the selectionprobability is greater than a preset selection probability threshold.When the selection probability is greater than a selection probabilitythreshold, step S5033 is performed and the processor sets the i^(th)lighting configuration as the currently optimal lighting configuration.Otherwise, step S5036 is executed by the processor to set the i−1^(th)lighting configuration as a currently optimal lighting configuration.

FIG. 7 illustrates a flow diagram of the fused imaging method accordingto an embodiment of the present disclosure. As shown in FIG. 7 , themethod is executed by a processor and includes determining, at stepS610, multiple lighting modes for capturing images of the target object,according to a preset multiple first optimal lighting configurations.The first optimal lighting configuration comprises a lightingconfiguration matrix with each value of the lighting configurationmatrix representing the working status of each independentlycontrollable LED and/or each pixel on the flexible screen. The workingstatus including at least one of: an off state, an on state, or a valueof brightness.

At step S620, the processor causes image capture component 120 tocapture images of the target object in each lighting mode to obtain aplurality of first images. At step S630, the processor generates atarget image of a target object based on the plurality of first images.

In another embodiment, the method may further cause the processor to: 1)obtain N lighting configurations from a preset set of lightingconfigurations by using N times of optimization selections, where N isan integer and N≥2; 2) determine scores of the N−1^(th) and the N^(th)lighting configurations; 3) determine a second optimal lightingconfiguration based on scores of the N−1^(th) and the N^(th) lightingconfigurations; and 4) determine a plurality of first optimal lightingconfigurations based on a second optimal lighting configuration. Thetarget image of the target object may be analyzed for surface defects inthe target object by control component 130. A processor, in controlcomponent 130, for example, may identify a surface defect from thetarget image and cause an indication to be provided to a user of alocation of a surface defect on the surface of the target object. Suchan indication may be provided by providing a display of the target imagepicture with a highlighted portion identifying a surface defect. Such anindication may be provided by causing light source component 120 toilluminate the surface defect for the user using the lighting source, alaser, or some other optical indication of a location of the surfacedefect on the target object.

The following examples are provided for illustrative embodiments of thisdisclosure:

Example 1. A fused imaging device, characterized in that the devicecomprises: a light source component which illuminates the target objectfor image capture where the light source component comprises a lightsource and a light housing, the light source including a plurality ofLEDs and/or flexible screens; an image capture component which capturesimages of the target object; a control component which is connected tothe light source component and the image capture component, the controlcomponent being configured to: control the light source component toilluminate the target object according to a plurality of preset firstoptimal lighting configurations, wherein the first of the plurality ofpreset optimal lighting configurations comprises a lightingconfiguration matrix, with each value of the lighting configurationmatrix representing the working status of each independentlycontrollable LED and/or each pixel on the flexible screen, the workingstatus being at least one of the three: off, on, or a value ofbrightness; control the image capture component to capture multipleimages of the target object to obtain multiple first images, under theillumination of the light source component; and generate a target imageof the target object based on the plurality of first images.

Example 2 may include the example of claim 1 and be furthercharacterized in that the control component is further configured to:obtain N lighting configurations from the preset set of lightingconfigurations by using N times of optimization selections, where N isan integer and N≥2; determine the scores of the N−1^(th) and the N^(th)lighting configurations; determine the second optimal lightingconfiguration based on the scores of the N−1^(th) and the N^(th)lighting configurations; and determine a plurality of first optimallighting configurations based on the second optimal lightingconfiguration.

Example 3 may include examples 1 and 2 and be further characterized inthat the device obtains N lighting configurations from the preset set oflighting configurations, by using N times of optimization selections,comprising; selecting any lighting configuration from the preset set oflighting configurations as the 1st lighting configuration; determiningthe 2nd lighting configuration based on the 1st lighting configurationand preset selection criteria; determining the currently optimallighting configuration based on the i−1^(th) and the i^(th) lightingconfigurations, with i being an integer and 2≤i≤N−1; and determine thei+1^(th) lighting configuration based on the currently optimal lightingconfiguration and the selection criteria.

Example 4 may include examples 1-3 and be further characterized in thatthe device determines the currently optimal lighting configuration basedon the i−1^(th) and the i^(t)h lighting configurations, comprising:determining the scores of the i−1^(th) and the i^(th) lightingconfiguration, respectively and when the score of the i^(th) lightingconfiguration is greater than the score of the i−1^(th) lightingconfiguration, setting the i^(th) lighting configuration as thecurrently optimal lighting configuration.

Example 5 may include examples 1-4 and be further characterized in thatthe device determines the currently optimal lighting configurationaccording to the i−1^(th) and the i^(th) lighting configurations,further comprises: when the score of the i^(th) lighting configurationis less than or equal to the score of the i−1^(th) lightingconfiguration, determining the probability of choosing the i^(th)lighting configuration as the currently optimal lighting configuration,based on the scores of the i^(th) and the i−1^(th) lightingconfigurations, and the selection round i; and when the selectionprobability is greater than the preset probability threshold, settingthe i^(th) lighting configuration as the currently optimal lightingconfiguration.

Example 6 may include any of examples 1-5 and be further characterizedin that the device determines the currently optimal lightingconfiguration according to the i−1^(th) and the i^(th) lightingconfigurations, further comprises: when the selection probability isless than or equal to the probability threshold, setting the i−1^(th)lighting configuration as the currently optimal lighting configuration.

Example 7 may include any of examples 1-6 and be further characterizedin that the device determines the score of the i−1^(th) lightingconfiguration, comprises: controlling the light source component toilluminate the target object according to the i−1^(th) lightingconfiguration; capturing an image of the target object by using theimage capture component to obtain the i−1^(th) image; determining thedefect area in the i−1^(th) image according to the preset defect area;determining the first average pixel values of the plurality of the firstpixels and the second average pixel values of the plurality of thesecond pixels, both in the i−1^(th) image; of which, the first pixelsare pixels inside the defect area in the i−1^(th) image, and the secondpixels are pixels outside the defect area in the i−1^(th) image;calculating the difference between the first and second average pixelvalues, and setting the ratio of this difference to a preset theoreticalmaximum difference as the score of the i−1^(th) lighting configuration.

Example 8 may include any of examples 1-7 and be further characterizedin that the light housing is approximately in the shape of ahemispherical shell, and the light source is arranged inside the lighthousing in configurations of cross-sections, geodesic dome, orhemispherical shell, such that the vertical and/or the horizontalincidence angle of the light from the light source is controllable.

Example 9 may include any of examples 1-8 and be further characterizedin that it generates a target image of the target object according tothe plurality of first images, using at least one of the followingmethods: determining, respectively, the optimal target pixels for eachpixel position, based on the characteristic value of each pixel at thesame pixel position in the plurality of first images, and generating atarget image of the target object according to the plurality of targetpixels; selecting a plurality of characteristic areas from the pluralityof first images according to preset selection criteria, and generating atarget image of the target object according to the plurality ofcharacteristic areas; and generating a three-dimensional image of thetarget object according to the plurality of the first images, andsetting this three-dimensional image as the target image of the targetobject.

Example 10 may include any of examples 1-9 and be further characterizedin that the method comprises: determining multiple lighting modes forcapturing images of the target object, according to the preset multiplefirst optimal lighting configurations, wherein the first optimallighting configuration comprises a lighting configuration matrix, witheach value of the lighting configuration matrix representing the workingstatus of each independently controllable LED and/or each pixel on theflexible screen, with the working status being at least one of thethree: off, on, or a value of brightness; in each lighting mode,capturing images of the target object to obtain multiple first images;and generating a target image of the target object based on theplurality of first images.

The above are only examples of embodiments of the present invention anddo not limit the scope of the patent protection of the presentinvention. Any equivalent transformation of structures and processes,made using the description and drawings of the present invention, ordirectly or indirectly applied to other related technical fields, aretherefore also included in the scope of patent protection of the presentinvention.

What is claimed is:
 1. A device, comprising: a light source; a camera;and a processor, wherein the light source includes a plurality of lightsdisposed at intercepts of each of a horizontal and vertical crosssection of a shell of the light source, the shell of the light sourcehaving a top and a bottom, wherein the plurality of lights are disposedat intercepts of each of the horizontal and vertical cross section ofthe shell of the light source and from the bottom of the shell to thetop of the shell.
 2. The device of claim 1, wherein the camera includesan image sensor.
 3. The device of claim 2, wherein the processor causesthe light source component to illuminate a target object.
 4. The deviceof claim 3, wherein the target object is illuminated according to aplurality of preset first optimal lighting configurations.
 5. The deviceof claim 4, wherein the plurality of preset first optimal lightingconfigurations used to illuminate the target object is determined basedon at least one second optimal lighting configuration.
 6. The device ofclaim 5, wherein the plurality of lights in the light source areindividually controllable by the control component.
 7. The device ofclaim 1, wherein the shell is a hemispherical shell.
 8. The device ofclaim 1, wherein the shell is a geodesic dome shell.
 9. The device ofclaim 8, wherein the geodesic dome shell comprises a plurality oftriangular surfaces which each include one or more of the plurality oflight sources.
 10. The device of claim 1, wherein the processor causesthe camera, which includes an image sensor, to capture a plurality ofimages of a target object under a plurality of lighting conditionsprovided by the light source component to generate a target image of thetarget object based on the plurality of images of the target object. 11.The device of claim 10, wherein the processor evaluates the target imageof the target object for surface defects in the target object.
 12. Thedevice of claim 11, wherein the processor provides an indication thatthe target object has no surface defects or an indication of a locationof a surface defect on the target object.
 13. The device of claim 1,wherein the plurality of lights are LEDS.
 14. The device of claim 1,wherein the plurality of lights is a plurality of pixels in a flexiblescreen.